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Nationwide, graduation rates at community colleges are discourag- 
ingly low. This randomized experiment provides evidence that gradu- 
ation rates can be increased dramatically. The City University of New 
York’s (CUNY) Accelerated Study in Associate Programs (ASAP) is 
a comprehensive, integrated, 3-year program that has an estimated 
1S percentage point effect on 3-year graduation rates, increases 
6-year graduation rates by an estimated 10 percentage points, and 
helps students graduate more quickly. Graduation effect estimates 
of this magnitude are exceptional in randomized experiments con- 
ducted in higher education, offering hope of what is possible when 
serving low-income students. (JEL H75, 123, 124, 128) 


ver the last 40 years the proportion of jobs that require postsecondary educa- 

tion has doubled, and labor market demands for educated workers are expected 
to continue to grow (Belfield and Bailey 2017; Carnevale, Jayasundera, and Gulish 
2016; Carnevale, Smith, and Strohl 2010; Chetty et al. 2017; Kena et al. 2016; 
Minaya and Scott-Clayton 2017; Ost, Pan, and Webber 2018; Scott-Clayton and 
Wen 2017). Simultaneously, community colleges have increased their share of post- 
secondary enrollees. In 2016-2017, they served 9 million students at nearly 1,000 
institutions, representing 39 percent of all US undergraduates!!!Despite providing 
unprecedented access to postsecondary education, rates of degree attainment remain 
disappointingly low. Among first-time, full-time students whose first postsecond- 
ary school is a 2-year public institution, only 22 percent earn a degree or certifi- 
cate within 3 years and only 42 percent earn a degree or certificate within 6 years 
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(National Student Clearinghouse 2017). These estimates are even lower for stu- 
dents entering community college requiring developmental (remedial) coursework 
(Bailey, Jeong, and Cho 2010). 

Numerous reforms have attempted to improve community college students’ rates 
of persistence and completion (Evans et al. 2017, Hatch 2016). These often include 
one or a few program components aimed at specific barriers to academic success, 
and they typically last one or two semesters. Although some of these programs have 
been found to improve students’ academic outcomes in the short term, few have 
been found to affect graduation rates substantially. 

In contrast, one program has been found to increase graduation rates consider- 
ably. This paper examines the effects of an uncommonly comprehensive, integrated, 
three-year program serving low-income community college students requiring 
remedial courses. The program requires students to attend college full time and 
encourages them to take remedial courses early, enroll in the summer, and graduate 
within three years. Students receive frequent advising, enhanced career services, 
and additional tutoring. Three forms of financial supports alleviate students’ finan- 
cial need: a tuition waiver, free use of textbooks, and a monthly transportation ben- 
efit, which is contingent on participation in key program services. Students can also 
enroll in courses with other program students in convenient schedules. 

This paper presents results from a randomized controlled trial evaluating this pro- 
gram’s impact on students’ academic progress and success. The program produces 
substantial positive impacts on full-time enrollment and credit accumulation, has 
an estimated 18 percentage point effect on 3-year graduation rates, increases 6-year 
graduation rates by an estimated 10 percentage points, and helps students graduate 
more quickly. To the authors’ knowledge, these are the largest intention-to-treat 
effect estimates among community college RCTs?) 


I. Background and Program Description 


Accelerated Study in Associate Programs (ASAP), created and implemented by 
the City University of New York (CUNY), is a comprehensive three-year program 
intended to help more students graduate within three years and help students grad- 
uate more quickly than they otherwise would. Launched in 2007 with funding from 
the New York City Center for Economic Opportunity, ASAP began at six commu- 
nity colleges and has now been implemented at nine CUNY institutions offering 
associate’s degrees. ASAP is designed to simultaneously address multiple potential 
barriers to student success. Below, we review the literature on barriers to college 
success for low-income students and then describe the CUNY ASAP intervention. 


?One notable exception is an evaluation of the effects of early college high schools, a program that targets 
high school students, but has also been found to increase two-year degree completion dramatically (see Berger 
et al. 2014). 
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A. Background and Barriers to Academic Success 


Community colleges, with their open admissions policies and comparatively 
low costs, represent a highly accessible pathway for low-income individuals to 
earn a postsecondary credential. As noted above, however, graduation rates from 
community colleges are very low. Moreover, a majority of students take longer 
to complete their degrees than is expected—that is, two years for an associate’s 
degree (Snyder, de Bray, and Dillow 2016). The consequences of this increased 
time to degree are considerable; the longer it takes a student to graduate, the more 
likely it is that she or he will encounter personal or financial difficulties that will 
make it necessary to drop courses, take time off, or stop college altogether. 

Scholars have identified a number of student-level challenges and institutional 
practices that help explain the low rates of persistence and completion at community 
colleges (for overviews, see Baum, Kurose, and McPherson 2013; Braxton 2000; 
Calcagno et al. 2008). While it is beyond the scope of this paper to review all the 
barriers described in the literature, we provide a brief overview of several nota- 
ble barriers that are directly addressed by CUNY ASAP. These include insufficient 
preparation for college-level work; a shortage and underutilization of student sup- 
port services; financial issues; and the competing demands of work, family, and 
school. These challenges are especially pronounced for low-income, academically 
underprepared students, such as the ones in this study. These students are less likely 
to enroll in school full time, which makes it more difficult to accumulate enough 
credits to stay on track for a timely graduation (Attewell, Heil, and Reisel 2012; 
Crosta 2014; Klempin 2014). 

Large proportions of low-income community college students are referred to 
remedial coursework, which is associated with poor college performance (Bailey, 
Jeong, and Cho 2010; Bound, Lovenheim, and Turner 2010; Chen and Simone 
2016). In fact, fewer than half of students who test into remedial courses complete 
the remedial sequence to which they are referred, let alone graduate (Bailey, Jeong, 
and Cho 2010). There is some evidence that ensuring academically underprepared 
students pass remedial courses early can improve postsecondary outcomes (Long 
and Boatman 2013, Scott-Clayton and Rodriguez 2012). A wide variety of reforms 
have been tried, but being academically underprepared remains a significant barrier 
to students’ academic success (Rutschow and Schneider 2011). 

Even before experiencing academic challenges, many low-income community 
college students struggle navigating an unfamiliar college environment. This chal- 
lenge may be particularly acute for first-generation college-goers, who are dispro- 
portionately represented at two-year colleges compared with four-year colleges 
(Deil-Amen 2011; Person, Rosenbaum, and Deil-Amen 2006). Upon enrolling 
in school, students may need help figuring out which courses to take and in what 
order, how to register for classes and apply for financial aid, and what resources are 
available to help make the transition to college (Bailey, Jaggars, and Jenkins 2015; 
Karp 2016; Kolenovic, Linderman, and Karp 2013). Community colleges are rarely 
able to support the kind of personalized and timely advising that students need 
(Bound and Turner 2007). The National Academic Advising Association estimates 
that the median caseload of an adviser at public two-year colleges is 441 students 


256 AMERICAN ECONOMIC JOURNAL: APPLIED ECONOMICS JULY 2019 


per adviser—severely limiting the amount of advising students receive. (At CUNY 
colleges, the caseload is estimated to be higher.) A fairly robust experimental liter- 
ature on enhanced academic advising interventions finds positive, although often 
modest, causal effects on students’ academic outcomes using a variety of modes of 
advising (for examples, see Avery, Howell, and Page 2014; Bailey et al. 2016; Barr 
and Castleman 2017; Bettinger and Baker 2014; Carrell and Sacerdote 2013, 2017; 
Evans et al. 2017; Oreopoulos and Petronijevic 2016; Scrivener and Weiss 2009). 

Finances also present a significant barrier to success for students (Denning 2017). 
Although tuition and fees at community colleges are about one-third the cost of pub- 
lic four-year colleges and universities, they still constitute a substantial investment 
for many low-income students, especially when opportunity costs are considered. In 
addition, financial aid sometimes does not cover the full cost of attendance, leaving 
students to struggle to afford necessities like textbooks or transportation. To cope, 
many students take on more work hours or enroll part-time, both of which cor- 
relate with reduced academic success (Horn, Berger, and Carroll 2004). Moreover, 
low-income students have to negotiate a complex financial aid system in order to 
receive aid (Bettinger et al. 2012). This issue is compounded by community colleges’ 
shortage of on-campus student support services, as noted above. Not only is there 
a correlation between student success and financial concerns, but multiple studies 
have produced experimental evidence that finance-related reforms can positively, 
although often modestly, influence students’ academic progress (Angrist et al. 2014, 
2016; Angrist, Lang, and Oreopoulos 2009; Angrist, Oreopoulos, and Williams 2010; 
Bettinger et al. 2012; Cohodes and Goodman 2014; Deming and Dynarski 2009; 
Goldrick-Rab et al. 2016; Mayer et al. 2015). 

Community college students, most of whom commute, have also been identified 
as less likely to identify with the college community than traditional undergradu- 
ates at four-year institutions (Tinto 1997). Colleges have tried many ways to inte- 
grate new students, including learning communities and student success courses 
to foster connections within the classroom (Engstrom and Tinto 2008; Rutschow, 
Cullinan, and Welbeck 2012; Weiss, Mayer et al. 2015; Weiss, Visher et al. 2015). 
Other strategies include interventions aimed at helping community college students 
enroll continuously—including in summer sessions (Attewell and Douglas 2014, 
Attewell and Jang 2013). Experimental evaluations of these approaches indicate that 
they can help students acclimate to college and some produce modest improvements 
on academic outcomes. 

Taken together, the numerous barriers that low-income students face contribute to 
the low success rates found in community colleges. While there is causal evidence 
about the efficacy of postsecondary interventions across the literature, it seems that 
short-term or light-touch interventions may not be robust enough to substantially 
improve a large proportion of students’ long-term outcomes. Only a few of the 
reforms that have been evaluated using a random assignment design have been found 
to increase graduation rates; those estimated increases are quite modest—4 percentage 
points or fewer—and many are statistically indistinguishable from 0 by the end of the 
evaluation (for examples, see Goldrick-Rab et al. 2012; Mayer, Patel, and Gutierrez 
2015; Weiss, Mayer et al. 2015). This may suggest that to dramatically increase 
graduation rates, it is necessary to implement more comprehensive programs that 
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address many barriers, and to offer those services for a longer time period. CUNY’s 
ASAP does just that. 


B. Program Description and Theory of Action 


CUNY ASAP is a multifaceted, integrated, three-year program that directly 
addresses many of the barriers to academic success discussed above. At the time of 
this study, the program comprised the following components: 


Student Responsibility and Messages.—ASAP required students to attend college 
full time each fall and spring semester that they were in the program and encouraged 
them to take courses during winter and summer sessions, both of which correlate 
with academic success. ASAP’s messages to students included emphasis on taking 
remedial courses early (to ensure that students mastered basic skills as soon as pos- 
sible) and graduating within three years. 


Student Support Services.—ASAP’s requirements and messages were comple- 
mented by a set of wraparound services intended to meet students’ academic and 
personal needs. Most centrally, each ASAP student was assigned to a dedicated 
adviser who provided frequent, comprehensive support. ASAP advisers were trained 
on a wide variety of academic and personal topics, including academic planning, 
balancing school with other responsibilities, accessing campus services, interact- 
ing with professors, staying on track to graduate, and dealing with personal issues. 
During the evaluation, advisers typically had caseloads of 60 to 80 students, and 
students were required to meet with their adviser twice per month. Comprehensive 
advising was intended to mediate issues for students, leading them to make bet- 
ter decisions and solve problems before they dropped out of school. Additionally, 
each college had an ASAP Career and Employment Specialist who students were 
required to meet with at least once per semester. These specialists were there to help 
students focus on career goals early in their schooling, and they continued to do so 
as they progressed. ASAP also required weekly tutoring for all students in remedial 
courses or on academic probation, with the goal of giving students a better chance 
of learning the material, passing the course, and moving to college-level work. All 
student support services were offered for the full three years of the program. 


Financial Supports —ASAP provided three forms of financial support to reduce 
students’ financial barriers to full-time attendance and lessen financial stress. The 
program provided a tuition waiver that covered any gap between federal and state 
financial aid and college tuition and fees. It also provided monthly MetroCards for 
use on New York City’s public transportation, contingent on participation in key 
program services (e.g., visiting an adviser two times per month), and free use of 
textbooks. These services were offered for up to three years. 


Structured Course Enrollment—ASAP provided blocked courses and consol- 
idated schedules throughout students’ first year in the program. Seats in courses 
were reserved for ASAP students with some courses scheduled back-to-back in 
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convenient blocks or offered as a set. The main goal of the blocked courses was 
to ensure that ASAP students could take some of their classes with other ASAP 
students, fostering a sense of community. Consolidated schedules were intended to 
allow students to make the most of their time on campus and more easily find a way 
to make enrolling full time in school fit around their other obligations. At the time 
of the evaluation, the program also included an ASAP seminar during students’ first 
few semesters (the specific number of semesters varied by college), covering topics 
such as goal-setting and study skills. An addition goal of blocked courses and the 
ASAP seminar was to increase students’ connections with each other and build a 
sense of community. 


Program Management.—ASAP was jointly administered by the centralized 
CUNY Office of Academic Affairs and the participating community colleges. 
Program staff members at each college tracked students’ participation in a home- 
grown database, and program directors as well as evaluation staff at CUNY central 
regularly reviewed student data and outcome trends to ensure the program was oper- 
ating as intended. ASAP operated with an uncommonly high level of monitoring 
and internal assessment compared with many college programs. 


Program Eligibility —To be eligible for the program at the time of this study, stu- 
dents had to meet several criteria: had family income below 200 percent of the fed- 
eral poverty level or were eligible for a Pell grant, required 1 or 2 remedial courses, 
had earned 12 or fewer credits before entering the program, were New York City res- 
idents, were willing to attend college full time, and were in ASAP-eligible majors.) 
Program staff checked students’ eligibility prior to their joining the program, and for 
the purposes of this study (described below), prior to random assignment. 


II. Evaluation Design 
A. Identification Strategy, Impact Model Specifications, and Student Sample 


We used a multi-site individual random assignment research design to identify 
the causal effects of the opportunity to participate in ASAP. The evaluation’s ana- 
lytic sample includes 896 students who were eligible for the program, signed an 
informed consent form, and agreed to participate in the evaluation. Two cohorts 
of students were randomly assigned, one prior to the spring and fall semesters in 
2010, at three of the six community colleges that ran ASAP at that time. Random 
assignment was controlled by the research team. Students were assigned either to 
the program group, whose members had the opportunity to participate in ASAP, or 
to the control group, whose members had the opportunity to participate in all of 
their colleges’ programs and services, just not ASAP. Students had an equal chance 
of being assigned to the program group or the control group. Over 95 percent of 
program group members were exposed to at least some portion of the program 


3 Approximately five majors were excluded from ASAP (e.g., nursing) because their requirements (e.g., 
prerequisites and clinical/practicum) make graduating within three years difficult or impossible. 
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(Scrivener, Weiss, and Sommo 2012}, thus we focus analyses on the effect of the 
program offer (i.e., ae In total, 903 students were randomly assigned. 
Seven students are not included in any analyses because they withdrew from the study 
or their consent form was unrecovered, leaving an analytic sample of 896 students.) 


B. Impact Model Specifications 


To obtain a regression-adjusted estimate of the causal effect of the opportunity to 
participate in ASAP, we use the following estimation model: 


(1) Y; = BZ, + yRB; + 6X; +4 «;, 


where Y; is the outcome for student i; Z; equals one if student i was assigned to 
treatment and zero otherwise; RB; is a vector of five random assignment block indi- 
cators (one for each unique college by cohort in the study); and X; is a vector of 
baseline characteristics included in the model because of their potential to improv 
the precision of the impact estimator (Bloom, Richburg-Hayes, and Black 2007) 
Robust (Huber-White) standard errors are used in all analyses. Analyses for all aca- 
demic outcomes at all time points presented in Section IV include all 896 students, 
unless otherwise noted. Analyses from a one-year follow-up survey (presented in 
Section III) include survey respondents only,’ Sensitivity analyses are conducted 
excluding the vector of baseline characteristics. A detailed discussion of all data 
sources is provided in Scrivener et al. (2015, 14-16). 


C. Sample 


additional baseline characteristics are shown in Appendix Table A 1). 

Similar to the gender breakdown at community colleges nationwide, more than 
half of sample members are women.® Reflecting the student population served at 
these three colleges, the sample is racially diverse, with no racial majority. Over 
three-quarters of the sample are black or Hispanic. Most sample members were of 


Table 1 |reports select characteristics of the analytic sae at baseline (several 


*For those interested in the estimated effect of the treatment offer on those individuals who received at least 
some portion of the treatment, take our intention-to-treat effect estimates and divide by 0.95, as described by 
Bloom (1984). 

>The overall attrition rate is 0.78 percent and the rate of differential attrition is 1.11 percentage points. By 
“withdrew from the study” we do not mean “withdrew from the program”—this refers to students who revoked 
consent for data collection. 

©Baseline characteristics come from the pre-random assignment baseline survey described in Scrivener et al. 
(2015). Characteristics were selected primarily based on prior evidence of a relationship to academic completion; 
for example, see Weiss et al. (2015), Mayer et al. (2015), who explore this issue. We include gender, race, age, has 
any children, single parent, working, depends on parents for more than half of expenses, first in family to attend 
college, and earned a high school diploma. For each baseline characteristic, a missing indicator is also included in 
the model as described by Gerber and Green (2012). Puma et al. (2009) shows that this approach does not introduce 
bias in randomized trials. 

7 The overall survey response rate was 83 percent. Program group students responded at a rate of 85.1 percent 
and control group students at a rate of 80.8 percent. For more information, see Appendix B of Scrivener et al. (2015). 

8Source: US Department of Education, National Center for Education Statistics, 2003-04 Beginning 
Postsecondary Students Longitudinal Study, Second Follow-up (BPS:04/09) 
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TABLE 1—SELECT CHARACTERISTICS OF SAMPLE MEMBERS AT BASELINE 


Control Estimated 


Characteristic (percent) mean difference | Observations 
Female 60.2 3.7 896 
3.2) 
Race/ethnicity 
Asian or Pacific Islander 79 —0.8 863 
1.8 
Black 35.9 —3.2 863 
3.2 
Hispanic 42.7 1.5 863 
3.3 
White 9.6 isd 863 
2.0 
Other* 4.0 1.4 863 
1.4 
Age 
19 or younger 57.5 —0.8 896 
3.3 
20-23 years 23.8 —2.8 896 
2.8 
24 or older 18.7 3.6 896 
2a 
Diplomas/degrees earned? 
High school diploma 735 0.6 878 
3.0 
GED certificate 21.1 —0.5 878 
Dell 
Occupation /technical certificate 6.6 —2.1 878 
1.6 
Other 1.8 —0.3 878 
0.9 
None 6.2 —0.2 878 
1.6 
Number of remedial courses needed 
None 2D —0.7 896 
0.9 
One 32.1 5.8 896 
32) 
Two 46.5 —3.1 896 
3.3 
Three or more 74 —0.8 896 
1.7 
Missing 11.7 —1.2 896 
2.1 
(continued) 


“traditional” college age, which is typical of CUNY community college students. 
While most students were of traditional college age, around 36 percent of the sam- 
ple had 1 or more of the following “nontraditional” characteristics: were 24 years 
old or older, worked 35 or more hours per week, had children, or did not receive 
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TABLE 1—SELECT CHARACTERISTICS OF SAMPLE MEMBERS AT BASELINE (continued) 


Control — Estimated 
Characteristic (percent) mean difference Observations 


Subject of remedial need 


No need 22, —0.7 896 
0.9 
English only 15.3 —0.7 896 
2.4 
Math only 44.9 5.0 896 
3.3 
English and math 25.8 —2.4 896 
2.9 
Missing 11.7 —1.2 896 
24 
College at random assignment 
College A 19.1 —0.3 896 
0.0 
College B 44.5 0.5 896 
0.0 
College C 36.4 —0.3 896 
0.0 
Cohort 
Spring 2010 35.7 1.5 896 
0.0 
Fall 2010 64.3 —1.5 896 
0.0 
Currently employed 32.6 —2.5 852 
3.2 
First in family to attend college 32.5 —4.3 852 
3.2) 
Nontraditional‘ 33.8 3.7 888 
3.2 


Notes: Missing values are only included in variable distributions for characteristics with more than 5 percent of the 
sample missing. Estimated differences are adjusted by random assignment blocks only. Standard errors are reported 
in brackets. 

* The “Other” category includes students who self-identified as Native American, Alaska Native, or other. 

> Distributions do not add to 100 percent because categories are not mutually exclusive. 

© This includes students who were enrolled in high school at study intake. 

4 Nontraditional students are defined as those who were 24 or older, worked 35 or more hours per week, had 
children, or did not receive a high school diploma and were not enrolled in high school at the time of random 
assignment. Students are listed as nontraditional if they fit any of these characteristics. Students are considered to 
be missing in the nontraditional category if they were missing data on 2 or more of these characteristics and have no 
other nontraditional characteristic; less than 5 percent of the study sample is missing in the nontraditional category. 


Sources: MDRC calculations from a Baseline Survey and CUNY Institutional Research Database (IRDB) 


a high school diploma and were not enrolled in high school at the time of random 
assignment. Finally, based on CUNY-administered exams, nearly the entire ana- 
lytic sample required some form of remedial education (an eligibility requirement), 
and more than three-quarters required remedial math. In other words, the program 


°For 11 percent of the sample this information is considered “missing” because those students do not have 
pre-random assignment CUNY Assessment Test data in all subject areas. 


262 AMERICAN ECONOMIC JOURNAL: APPLIED ECONOMICS JULY 2019 


served many at-risk students. Also shown in Table 1 and Appendix| Table A1J the 
background characteristics of program and control group students were very similar 
at the outset of the study, as is expected in a randomized experiment, 


Ill. Program Implementation and Service Contrast 


Implementation research was conducted periodically throughout the first three 
years of the study. Overall, ASAP was well implemented and the difference between 
ASAP and regular college services available to the study’s control group was sub- 
stantial. We briefly discuss each program component, in turn—for details see 
Scrivener et al. (2015). 


Messages.—In a 12-month follow-up student survey, | program group students 
reported significantly different experiences in terms of ASAP’s planned messaging. 
Approximately 31 percent of control group students reported often or very often 
hearing college faculty /staff speak about the importance of taking developmental 
courses early and obtaining an associate’s degree within 3 years. Program group 
members were 16 and 38 percentage points more likely to report messaging on these 
topics, respectively (p < 0.001 in both cases). 


Student Support Services——Program group students reported vastly different 
experiences compared with control group students on their usage of support ser- 
vices. [Table 2| presents survey findings regarding students’ self-reported use of 
support services. There are large differences in participation in advising, career ser- 
vices, and tutoring outside of class, as well as large differences in the total number 
of times students partook of these activities. For instance, 95 percent of program 
group students reported contact with advising, compared with 80 percent of control 
group students—but even more telling, program group students reported an average 
of 21 and 17 contacts per semester in their first year, compared with control group 
students’ who reported an average of 4 and 2 contacts per semester. This pattern car- 
ries through for career services and tutoring, though the differences in the number 
of contacts are less dramatic. 


Financial Supports.{Table 3] presents students’ experiences with financial 
supports. Recall that CUNY ASAP offers students multiple financial supports, 
including a last-dollar scholarship for tuition and fees, free use of textbooks, and 
a monthly MetroCard for use on public transportation. We found large differences 
in students’ reported receipt of transportation assistance (panel A) and textbooks 
(panel B), indicating a dramatic service contrast. Interestingly, panel C indicates 


10 We analyzed whether all baseline characteristics jointly predict research group status, using logistic regres- 
sion. A likelihood ratio test was performed to assess the joint power of the baseline variables to predict treatment 
status. The test yields a p-value of 0.89, reinforcing that there are not systematic differences in the baseline charac- 
teristics of program and control group members (Scrivener et al. 2015). An F-test for joint balance yields the same 
conclusion (p = 0.88). 

'l The evaluation included 1 student survey administered 12 months after study entry; program and control 
group members’ service usage during later years was not measured. 
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TABLE 2—USE OF STUDENT SUPPORT SERVICES IN YEAR | 


Estimated effect 


Control mean Covariate adjusted Unadjusted Observations 
Panel A. Advising 
Ever contacted (percent) 80.4 15.0 14.6 739 
2.4 2.4 
Number of times 
First semester 37 17.5 17.4 718 
(6.1) 1.4 1.3 
Second semester 2.0 14.5 14.6 718 
(4.0) 1.0 1.0 
Panel B. Career/employment services 
Ever contacted (percent) 28.9 50.0 50.6 736 
3.2 3.2 
Number of times 
First semester 1.0 3.8 3.8 719 
(3.5) 0.4 0.4 
Second semester 0.6 3.6 3.5 721 
(2.7) 0.4 0.4 
Panel C. Tutoring outside of class 
Ever contacted (percent) 39.4 33.3 34.2 736 
3.4 3.4 
Number of times 
First semester 4.0 8.5 9.0 724 
(11.4) 1.1 1.1 
Second semester 25 7.6 8.1 730 
(8.7) 1.0 1.0 


Notes: Covariate adjusted estimates are adjusted by random assignment blocks and select baseline characteristics 
(see section titled “Impact Model Specifications” for details). Unadjusted estimates are adjusted by random assign- 
ment blocks only. Standard deviations for continuous outcomes are reported in parantheses. Standard errors are 
reported in brackets. 


Source: MDRC calculations from the MDRC student survey 


that relatively few students (11 percent or less per semester) required a tuition 
waiver from the program. This is because the vast majority of students received 
sufficient federal and state need-based financial aid to fully cover their tuition and 
fees. 


Structured Course Enrollment.—In addition to the student survey, we analyzed 
transcript data to assess the implementation of course enrollments in the ASAP sem- 
inar and blocked courses in their first year. Implementation of these components 
varied by college (some colleges had high compliance in these courses), and at all 
colleges, a majority of students enrolled in courses with a significant concentration 
of ASAP students, one goal of the program. 

In sum, CUNY ASAP was well implemented and students offered ASAP had a 
vastly different experience in college compared to the status quo. We now turn to 
the program’s estimated impacts on students’ academic outcomes—effects that are 
caused by the large differences in experiences reported above, which result from 
students being offered ASAP. 
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TABLE 3—FINANCIAL SUPPORTS 


Estimated effect 


Control Covariate 
mean adjusted Unadjusted Observations 
Panel A. Transportation 
Received financial support, year 1 (percent) TA 86.7 86.3 729 
[1.9] [1.9] 
Panel B. Textbooks 
Received all free of charge, year 1 (percent) 5.7 70.7 71.2 729 
[2.5] [2.5] 
Panel C. Tuition waiver 
Received ASAP waiver (percent) 
Semester 1 N/A 8.9 451 
Semester 2 N/A 93 451 
Semester 3 N/A 10.4 451 
Semester 4 N/A 10.6 451 
Semester 5 N/A 4.9 451 
Semester 6 N/A 3.3 451 


Notes: In panels A and B, covariate-adjusted estimates are adjusted by random assignment blocks and select base- 
line characteristics (see section titled “Impact Model Specifications” for details). Unadjusted estimates are adjusted 
by random assignment blocks only. Standard errors are reported in brackets. Panel C presents program group out- 
come levels. 


Sources: MDRC calculations from the MDRC student survey (panels A and B) and the tuition waiver analysis per- 
formed by CUNY ASAP office (panel C) 


IV. Program Effects on Academic Outcomes 


This section presents estimates of the effect of the opportunity to participate 
in ASAP on students’ academic outcomes during the six years after they entered 
the evaluation—three years when ASAP services were offered to program group 
members (program semesters) and three years after the program was complete 
(post-program semesters). We proceed with estimates of ASAP’s effects on enroll- 
ment, credit accumulation, degree completion, and transfer, followed by estimates 
of effects for select subgroups of students. 


A. Main Effects on Enrollment 


Figure 1 |(Appendix|Table A2) presents enrollment rates (y-axis) at any college 


throughout the United States during the 6 years, or 12 semesters (x-axis), after ran- 
dom assignment. (Appendix/Table A3) shows full-time enrollment rates at 
any CUNY two- or four-year college.!* In all figures, squares and circles represent 
the regression-adjusted average when assigned to the program group and control 
group, respectively, and diamonds represent the estimated effect of the program 
offer, including a 90 percent confidence interval. The full sample is included at all 


'2Enrollment at and degree receipt from CUNY institutions is based on data from CUNY’s Institutional 
Research Database (IRDB); these are supplemented with enrollment at and degree receipt from non-CUNY institu- 
tions, using data from the National Student Clearinghouse. Credit accumulation measures are created using IRDB 
data only. Detailed information on data sources is provided in Scrivener et al. (2015). 
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FiGurE 1. ENROLLMENT AT ANY COLLEGE BY SEMESTER 


Notes: Estimates are adjusted by random assignment blocks and select baseline characteristics (see section titled 
“Impact Model Specifications” for details). Enrollment is based on courses that students are enrolled in at the end 
of the add/drop period. 


Sources: MDRC calculations from CUNY Institutional Research Database (IRDB) and National Student 
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FiGuRE 2. FULL-TIME ENROLLMENT AT ANY COLLEGE BY SEMESTER 


Notes: Estimates are adjusted by random assignment blocks and select baseline characteristics (see section titled 
“Impact Model Specifications” for details). Enrollment is based on courses that students are enrolled in at the end 
of the add/drop period. 


Source: MDRC calculations from CUNY Institutional Research Database (IRDB) 
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time points. Later, in Section IVE, enrollment is broken out by college type (two- 
and four-year colleges). 


Enrolled at Any College.—During the three years when program group students 
were offered ASAP services, the program reduced dropout rates. In semesters 2 
through 6, ASAP increased enrollment by an estimated 10, 7, 8, 6, and 4 percentage 
points, respectively. 

Because of ASAP’s large effects on graduation rates, starting in semester five 
(see below), interpreting enrollment rates in later semesters is complicated. Whereas 
early non-enrollment generally represents the negative outcome of students drop- 
ping or temporarily stopping out, later non-enrollment can also reflect students’ 
having already achieved their goal of a terminal degree. With that in mind, during 
all 6 post-program semesters the control group’s enrollment rates exceeded those 
of the program group, with impact estimates ranging between 2 and 5 percentage 
points—all negative effect estimates, none of which are statistically significantly 
different from 0 at conventional levels. This pattern of positive in-program effects 
followed by some degree of post-program control group catch-up plays out on mar- 
ginal credits earned and degree completion (see below). 

Interestingly, by the end of six years, on average, program and control group 
students had enrolled in a similar total number of terms. A simple average of the 
12 estimated effects on enrollment reveals that the average enrollment effect is just 
1.2 percentage points. As will be shown later, despite having enrolled in a similar 
number of semesters, program group members graduated at a much higher rate than 
control group members. This is in part because, as discussed next, ASAP increased 
full-time enrollment significantly,!> 


Full-Time Enrollment at a CUNY College.—Recall that full-time enrollment 
(attempting 12 or more credits per semester) is a requirement of ASAP. Moreover, all 
students who entered the evaluation agreed that they were willing to enroll full-time. 
Figure 2 shows that in the first semester immediately after random assignment, stu- 
dents offered ASAP were already substantially more likely to enroll full-time than 
their control group counterparts. In fact, during each of the 6 in-program semesters 
(3 years), ASAP had a positive effect on full-time enrollment, with effect estimates 
ranging from 6 to 20 percentage points. These results demonstrate that for a substan- 
tial number of students who currently enroll part time, if given the right combination 
of requirements, incentives, and supports, they will enroll full time. This finding is 
particularly important because nationally around half of community college entrants 


13 Although not shown in the tables, please note that the pattern of effect estimates on enrollment at any college 
is extremely similar to the pattern of estimated effects on enrollment at CUNY colleges (the largest difference is 
1.7 percentage points). In later semesters, however, the difference in enrollment levels (not estimated effects) at 
any college versus at CUNY colleges increases, peaking at a 7.6 percentage point difference in semester 10. We note 
this because data on full-time enrollment and credit accumulation are available at CUNY colleges only. With respect 
to estimated effects, it is likely that the full-time enrollment and credit accumulation results at CUNY colleges are 
similar to what they would be at any college, were the data available. However, with respect to outcome levels, the 
full-time enrollment and credit accumulation results at CUNY colleges are a lower bound of what we would expect 
at any college, were the data available. Fortunately, for the main outcome of interest—degree attainment—data are 
available at any college. 
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enroll part-time, and part-time attendance is associated with a decreased likelihood 
of succeeding in college (Attewell, Heil, and Reisel 2012). 

During the post-program semesters, there is not clear evidence of a meaningful 
effect (positive or negative) on full-time enrollment at CUNY colleges. However, 
unlike the pattern of effects for enrollment of any intensity level, a simple average 
of the 12 estimated effects on full-time enrollment reveals that the average full-time 
enrollment effect is 5.9 percentage points (not displayed in the table). In other 
words, overall, program group students enrolled full-time in more total semesters 
than did their control group counterparts. 

Enrollment and full-time enrollment are important indicators of academic prog- 
ress. However, degrees are not conferred based on enrollment—for that, students 
must accumulate credits, which we turn to next. 


B. Main Effects on Credit Accumulation 


and | (and Appendix Tables [A4|and depict credit accumulation at 
CUNY colleges during the 12 semesters after students were randomly assigned {4 
Figure 3 displays marginal credits earned, including credits earned in a particular 
semester only. Figure 4 plots cumulative credits earned, including all credits earned 
since the first semester after random assignment. 


Marginal Credits Earned at Any CUNY College.—Figure 3 shows that, as 
expected, average marginal credits earned decrease over time, a result that corre- 
sponds with some students dropping or stopping out and earning 0 credits. Also 
corresponding with enrollment trends, ASAP had a large positive effect on mar- 
ginal credit accumulation during each of the first 4 program semesters, with effect 
estimates ranging from 1.4 to 2.1 credits. During semesters 5 and 6, effect esti- 
mates remain positive, but they drop to 0.5 in both semesters,!> Throughout the 
three post-program years, program effect estimates on marginal credits earned 
hover around zero and are not statistically distinguishable from zero at conven- 
tional levels. 


Cumulative Credits Earned at Any CUNY College.—The depiction in Figure 4 
of average cumulative credit accumulation is illuminating because it is cumulative 
credits earned that lead to a degree. Throughout the six-year follow-up period, stu- 
dents offered ASAP earned significantly more cumulative credits than their control 
group counterparts. The gap between the 2 groups peaks at 8.2 credits in semester 7, 
before dipping to an estimated 7.0 credit effect after 6 years. To put this effect in 
context, 7.0 credits represent a 13 percent increase over the control group base of 
55.6 credits, or 12 percent of the 60 college-level credits required to earn an associ- 
ate’s degree. Relative to ITT effect estimates on credit accumulation in other higher 
education experiments, ASAP’s effect estimates are substantial. 


'4 Credits include remedial and college-level credits. 
'5 p-values on the effect estimates in semesters 1 through 4 are all <0.001. In semester 5, p = 0.236. In 
semester 6, p = 0.200. 
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Notes: Estimates are adjusted by random assignment blocks and select baseline characteristics (see Section IIB for 
details). 


Source: MDRC calculations from CUNY Institutional Research Database (IRDB) 
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FIGURE 4. CUMULATIVE CREDITS EARNED AT CUNY COLLEGES 


Notes: Estimates are adjusted by random assignment blocks and select baseline characteristics (see section titled 
“Impact Model Specifications” for details). Measures of cumulative credits earned exclude courses that are passed 
more than once. 


Source: MDRC calculations from CUNY Institutional Research Database (IRDB) 
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FIGURE 5. DEGREE COMPLETION AT ANY COLLEGE 


Notes: Estimates are adjusted by random assignment blocks and select baseline characteristics (see section titled 
“Impact Model Specifications” for details). Degree receipt is cumulative. Those who earned a degree in an earlier 
semester are counted as having a degree in subsequent semesters. 


Sources: MDRC calculations from CUNY Institutional Research Database (IRDB) and National Student 
Clearinghouse data 


Does this increase in enrollment and credit accumulation translate into degree 
receipt? That is considered next. 


C. Main Effects on Degree Receipt 


ASAP’s explicit goal is to get more students to graduate and to graduate more 


quickly, Figure 5 (and Appendix}Table A6) presents degree completion rates at any 
college, including two- or four-year degrees at CUNY or non-CUNY colleges. 


Degree Receipt at Any College.—As early as the end of the fourth semester after 
random assignment, ASAP had a positive, statistically significant, and practically 
meaningful effect on degree completion. By the end of the 3-year program, program 
group members graduated at a rate of nearly 40 percent and control group members 
graduated at a rate of 22 percent, for an estimated effect of around 18 percentage 
points (p < 0.001). A 90 percent confidence interval on this effect estimate ranges 
from 13 to 23 percentage points; thus, we are confident that the true effect is quite 
large. To our knowledge, 18 percentage points is the largest ITT effect estimate on 
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Notes: Earned a degree estimates are not regression-adjusted. Average semesters to earn degree is calculated as the 
average number of semesters it took for the first X percent of degree earners to earn their first degree. 


Sources: MDRC calculations from CUNY Institutional Research Database (IRDB) and National Student 
Clearinghouse data 


3-year graduation rates in any large-scale randomized experiment in higher educa- 
tion to date. 

As can be seen on the right half of Figure 5, in the 3 years (semesters 7 through 
12) following program completion, the gap in graduation rates between program 
and control group members begins to narrow. After 6 years, ASAP’s estimated effect 
on earning a degree was 10 percentage points (p < 0.01). The narrowing of the gap 
in post-program semesters implies that part of the 18 percentage point impact on 
3-year graduation rates was a result of ASAP getting students who would have grad- 
uated in the absence of the program to graduate more quickly. ASAP’s remaining 
10 percentage point impact on 6-year graduation rates likely reflects a combination 
of some students who would have graduated beyond 6 years and some students who 
would not have graduated at all without ASAP. 


Time to Degree.—+Figure 6 |provides another perspective on the extent that 
ASAP helps students graduate more quickly. In this exhibit, the y-axis represents 


average-time to degree among degree earners. The x-axis denotes quantiles, or the 
first X percent of students to have earned a degree!” Each point on the graph depicts 


'©One notable exception is a study of Early College High Schools, which finds a similarly large effect estimate 
(Berger et al. 2014). 

'7 Analyses of the effect of a program on students’ average time to degree are complicated by the fact that not 
all students earn a degree. For example, a comparison of the average time to degree among degree earners at a par- 
ticular follow-up time point is subject to selection bias. A comparison of the average time to degree comparing all 
program group members to all control group members is not possible because there is no value for those who did 
not earn a degree. Figure 6 overcomes these challenges by considering the program’s effects on the distribution of 
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the average time to degree at a given quantile. The vertical gaps between the two 
lines represent ASAP’s estimated effect in the average time to degree at a given 
quantile in the distribution. For example, the 41 percent of control group members 
who earned a degree within the 6-year follow-up period averaged 6.8 semesters to 
do so. In comparison, the first 41 percent of program group members who earned 
a degree averaged under 4.9 semesters to do so—nearly | whole year (2 semes- 
ters) earlier than the control group. ASAP clearly shifts the distribution of time to 
degree—an accomplishment that has important potential benefits in the labor mar- 
ket and elsewhere. 

A natural additional question is, what percentage of students did ASAP cause 
to earn a degree who otherwise would not have earned one? This is difficult to 
determine with certainty at this time point, but we offer relevant context. First, at 
the end of the follow-up period, many students are still progressing through school: 
24 percent of the program group and 27 percent of the control group were enrolled 
during the final semester of follow-up (see Appendix Table A2). Initially, this could 
lead one to believe that many additional degrees could be conferred in the future, 
and estimated effects on degree completion may change substantially. However, 
most students who enrolled in the final semester of follow-up already had earned a 
degree. In fact, only 6.5 percent of all program group members and 10.3 percent of 
all control group members were enrolled during semester 12 and had not yet earned 
any degree.!® Also, only 1.2 percent of all program group members and 2.3 percent 
of all control group members earned their first degree at the end of semester 12. The 
small number of first degrees conferred in semester 12 demonstrates the plateauing 
of degree receipt that is also seen in Figure 5. Together, this information suggests 
we should not expect a large number of new degrees to be earned in either research 
group in the coming semesters. Thus, while ASAP’s estimated effect on earning any 
degree may drop a bit in upcoming semesters, it appears unlikely that it will drop 
a lot, implying that, overall, ASAP likely causes some students to earn a degree 
who would not have done so in the absence of the program. Planned longer term 
follow-up will provide an empirical estimate. 


the outcome, an approach similar to quantile regression (e.g., see Friedlander and Robins 1997; Heckman, Smith, 
and Clements 1997; Abadie, Angrist, and Imbens 2002). Randomization allows unbiased estimation of the effect of 
a program on a distribution of an outcome. 

The approach we take is also similar to the nonparametric approach used by Angrist, Bettinger, and Kremer 
(2006) to estimate the effect of winning a voucher at different points in the cumulative distribution of test scores in 
the face of differential missing outcome data. However, we are in a far simpler situation because they had to make 
assumptions about missing test scores, which, in theory, could be located anywhere in the outcome distribution. In 
contrast, our missing data (due to time-censoring) is for individuals who have not yet earned a degree within six 
years. We know that the only possible values for time-to-degree for these nondegree earners are greater than six 
years or never. Thus, in the control group, they all fall above the forty-first quantile, and in the program group they 
all fall above the fifty-first quantile. Consequently, we plot the empirical distribution of the outcome up through the 
known quantiles without making a monotone treatment response assumption or a rank-preservation assumption, as 
required in Angrist, Bettinger, and Kremer (2006). To avoid imputation of unknown points on the empirical cumu- 
lative distribution, we focus discussion on quantiles where data are available for the program and control groups. 

'8 The 3.8 percentage point difference has a p-value of 0.042. 
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FIGURE 7. ENROLLMENT AT TWo- AND FourR- YEAR COLLEGES 


Notes: Estimates are adjusted by random assignment blocks and select baseline characteristics (see section titled 
“Impact Model Specifications” for details). Enrollment is based on courses that students are enrolled in at the end 
of the add/drop period. 


Sources: MDRC calculations from CUNY Institutional Research Database (IRDB) and National Student 
Clearinghouse data 


D. Main Effects on Transfer to and Graduation from Four-Year Colleges 


While transfer to a four-year college is not the primary aim of ASAP, the pro- 
gram’s large effects on earning an associate degree, coupled with its intensive advis- 
ing services, has the potential to yield increased enrollment in and attainment at 
four-year colleges.|Figure 7] (and Appendix|Table A7) displays enrollment rates at 
any colleges throughout the United States, separated into two-year and four-year 
colleges (Figure 1 focused on enrollment at any college). 


Enrollment at Two- and Four-Year Colleges.—During the first four semesters 
of the program, ASAP increased the enrollment rate at two-year colleges. Then, 
because ASAP students earned associate’s degrees sooner than their control group 
counterparts, they enrolled at four-year colleges sooner too (semesters five through 
eight). During semesters six through nine control group enrollment at two-year 
colleges surpassed program group enrollment at two-year colleges, allowing the 
control group to start catching up with respect to progress and completion of associ- 
ate’s degrees. This catch-up occurred because far more program group members had 
already earned an associate’s degree (semesters six through nine). Similarly, toward 
the end of the follow-up period, the control group began catching up with respect to 
enrolling at four-year colleges. 
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FiGuRE 8. BACHELOR DEGREE COMPLETION AT ANY COLLEGE 


Notes: Estimates are adjusted by random assignment blocks and select baseline characteristics (see section titled 
“Impact Model Specifications” for details). Degree receipt is cumulative. Those who earned a degree in an earlier 
semester are counted as having a degree in subsequent semesters. 


Sources: MDRC calculations from CUNY Institutional Research Database (IRDB) and National Student 
Clearinghouse data 


Degree Receipt at Two- and Four-Year Colleges._Figure 8 [presents bachelor 
degree completion at any college throughout the United States. Appendix |Table A8| 
presents degree completion at any college throughout the United States separated 
by associate’s and bachelor’s degrees (unlike Figure 5, which does not distinguish 
between these degrees). The most notable new finding is that there is some evidence 
(in semesters eight through ten) that ASAP had a small, positive effect on bachelor’s 
degree completion. This finding aligns with the program’s effects on enrollment at 
four-year colleges, which were largest in semesters five through seven. Notice that 
the estimated effect on earning a bachelor’s degree is near zero by the end of the 
follow-up period. This suggests that ASAP helped a small proportion of students 
earn a bachelor’s degree more quickly than they would have otherwise, but we do 
not find evidence that ASAP caused students who otherwise would not have earned 
a bachelor’s degree to do so. 


E. Subgroup Effects 
The above findings show that ASAP, on average, has a positive effect on earning 


a degree as of the three- and six-year marks. We conducted analyses to explore 
if these findings hold up for a variety of subgroups of students. These analyses 
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are considered exploratory (i) due to limited statistical power to detect differential 
effects; (ii) because of no clear, strong, directional hypotheses why ASAP’s effects 
would vary by measured characteristics; (iii) because of no prior empirical evi- 
dence that ASAP’s effects vary by observed characteristics (a condition suggested 
by Bloom and Michalopoulos 2013); and (iv) to reduce the multiple testing prob- 
lem (Schochet 2009). These analyses may nonetheless be fruitful in suggesting that 
ASAP is effective for a wide variety of student types. Moreover, should evidence 
of differential effects arise, this information may be used to generate hypotheses 
that could then be rigorously tested in the ASAP demonstration project in Ohio 
(described later) or other future evaluations of ASAP or ASAP-like programs. 

[Tables 4and|5]present three- and six-year graduation rates at any college, by sub- 
group. We discuss three-year rates first because the explicit goal of ASAP is to get 
students to graduate during the three-year program. For all subgroups explored, esti- 
mated effects on earning any degree at the three-year mark are large and positive,!? 
In other words, the evidence suggests ASAP is benefiting students with a variety of 
measured background characteristics. 

Table 5, which provides six-year degree results by subgroup, tells a fairly similar 
story. The key differences are that the effect estimates are smaller across the board 
(as expected given the main findings) and that more of the positive effect estimates 
for subgroups are not statistically distinguishable from zero, which is unsurprising 
given the relatively small sample size for each subgroup. Generally, it appears that 
ASAP is having positive effects for a variety of subpopulations of students. 

For the race subgroup, at the 3-year mark there is evidence of variation in pro- 
gram effects (p = 0.040), with very large positive effect estimates for black and 
white students, compared with the relatively smaller (but still large) positive effect 
estimates for Hispanic students. The effect estimate for Hispanic students is near 
zero at the six-year mark. It is possible that, compared with other students, Hispanic 
students respond differently to ASAP, given that their needs and barriers to success 
may be different. However, CUNY’s internal propensity score matching analyses of 
ASAP’s effects (including additional cohorts and all implementing colleges), do not 
find a similar pattern (Strumbos and Kolenovic 2016). Moreover, we examined 10 
subgroups and found one p-value below 0.10—just what we would expect due to 
chance. Consequently, at this point we are cautious about over interpreting this result. 


V. Discussion 


CUNY ASAP’s estimated effects on 3- and 6-year graduation rates, 18 
and 10 percentage points, respectively, are exceptionally large. To put this 
study and these findings in context, consider the overall dearth of causal evi- 
dence on the effectiveness of strategies that substantially improve graduation 


'Tn three instances (white students, students without a high school diploma at baseline, and students ran- 
domly assigned at college A) the p-value for the subgroup’s estimated effect on three-year degree completion 
was above 0.10, a conventional level of statistical significance. For white students, the effect estimate is very 
large, but the sample included only 86 white students. For students without a high school diploma, the effect 
estimate was 9 percentage points, which is also quite large. At college A, the effect estimate was 6 percentage 
points, the smallest of all effect estimates. 
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TABLE 4—THREE- YEAR DEGREE COMPLETION AT ANY COLLEGE (BY SUBGROUP) 


Control Estimated p-value 
mean effect Observations difference in effects 
Gender 0.790 
Female 24.6 17.1 556 
4.0 
Male 18.1 18.8 340 
4.9 
Race 0.038 
Black 18.2 27.4 296 
5,3 
Hispanic 23.0 9.4 376 
4.6 
White 26.8 17.1 86 
[10.1 
Age 0.338 
19 or younger 24.6 16.5 512 
4.2 
20 to 23 17.9 13.7 201 
6.7 
24 or older 19.3 27.0 183 
TA 
Earned high school diploma at baseline* 0.129 
Yes 22.9 19.8 669 
3.6 
No 19.5 9.1 227 
6.0 
Number of remedial courses needed 0.420 
1 or fewer 27.5 21.7 331 
5.4 
2 or more 19.6 16.2 466 
4.1 
College 0.127 
College A 21.2 6.0 170 
6.8 
College B 217 21,5 401 
4.6 
College C 22.8 21.6 325 
5.3 
Cohort 0.498 
Spring 2010 20.8 20.0 327 
5.2) 
Fall 2010 22.7 15.6 569 
3.8 


(continued) 


rates for community college students requiring developmental education. 
The US Department of Education’s What Works Clearinghouse (WWC), the 
largest repository of scientific evidence on the effectiveness of education 
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TABLE 4—THREE- YEAR DEGREE COMPLETION AT ANY COLLEGE (BY SUBGROUP) (continued) 


Control — Estimated p-value 
mean effect Observations difference in effects 
Employed at baseline 0.487 
Yes 22.5 20.2 267 
5.8 
No 22.1 15.4 585 
3.8 
First in family to attend college 0.615 
Yes 23.4 20.5 258 
6.0 
No 22.1 16.9 594 
3.8 
Traditional 0.948 
Yes 22.9 17.3 571 
3.9 
No 20.1 17.7 317 
5.1 


Notes: Estimates are adjusted by random assignment blocks and select baseline characteristics (see section titled 
“Impact Model Specifications” for details). Standard errors are repoted in braces. The “p-value difference in 
effects” column presents the p-value for a test of variation in treatment effects among the categories shown for each 
subgroup, based on the H statistic as described in Greenberg, Meyer, and Wiseman (1994). 

* Students shown as not having a high school diploma include those who earned no degrees, those who earned 
a General Educational Development (GED) certificate, and those who are missing degree information. Students 
shown as having a high school diploma are those who earned a high school diploma, an occupational or technical 
certificate, or another, unspecified higher degree. 


Sources: MDRC calculations from a Baseline Survey, CUNY Institutional Research Database (IRDB), and National 
Student Clearinghouse data 


programs, recently published a practice guide of strategies to help postsecond- 
ary students in developmental education (Bailey et al., 2016). After review- 
ing 25,697 studies, only 10 studies met the WWC evidence standards with or 
without reservations (primarily randomized trials). Of the guide’s six recom- 
mended practices, only three were supported by evaluations meeting the WWC 
evidence standards (one of the studies/practices is ASAP). This suggests that 
causal evidence on strategies that improve graduation rates for students placing 
into developmental education is desperately needed by college administrators 
and policymakers. 

Moreover, among all WWC postsecondary evaluations that meet their evidence 
standards, only six examine college graduation rates (Scrivener and Weiss 2009, 
Rodriguez-Planas 2010, Patel and Rudd 2012, Berger et al. 2014, Bettinger and 
Baker 2014, Scrivener et al. 2015). Across these studies, the estimated effects 
on graduation rates are 4 percentage points or smaller with one exception: the 
Early College High School initiative, a lauded model that has estimated effects 
on earning an Associate’s degree that are similarly as large as compared with 
ASAP. 


VOL. 11 NO.3 WEISS ET AL.: LONG-TERM RESULTS OF A RANDOMIZED TRIAL OF CUNY’S ASAP — 277 


TABLE 5—SIX-YEAR DEGREE COMPLETION AT ANY COLLEGE (BY SUBGROUP) 


Control —_ Estimated p-value 
mean effect Observations difference in effects 
Gender 0.722 
Female 43.3 11.4 556 
4.2 
Male 36.2 8.9 340 
5.5 
Race 0.046 
Black 35.7 20.6 296 
5.7 
Hispanic 42.1 7 376 
5.2 
White 39.0 15.0 86 
[11.7 
Age 0.489 
19 or younger 43.4 713 512 
45 
20 to 23 34.9 11.4 201 
74 
24 or older 38.6 17.8 183 
17 
Earned high school diploma at baseline* 0.414 
Yes 42.2 11.5 669 
3.9 
No 35.4 5.2 227 
6.7 
Number of remedial courses needed 0.784 
1 or fewer 47.1 10.6 331 
5.6 
2 or more 39.2 8.6 466 
47 
College 0.780 
College A 37.6 6.1 170 
7.8 
College B 40.4 12.1 401 
5.0 
College C 42.0 12.3 325 
5.6 
Cohort 0.708 
Spring 2010 38.4 11.9 327 
5.7 
Fall 2010 41.6 9.3 569 
4.1 
Employed at baseline 0.107 
Yes 39.1 18.1 267 
6.0 
No 41.8 6.3 585 
4.1 


(continued) 
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TABLE 5—SIx- YEAR DEGREE COMPLETION AT ANY COLLEGE (BY SUBGROUP) (continued) 


Control — Estimated p-value 
mean effect Observations difference in effects 
First in family to attend college 0.831 
Yes 39.4 11.8 258 
6.4 
No 41.4 10.2 594 
4.1 
Traditional 0.202 
Yes 43.2 6.6 S71 
4.2 
No 35.6 15.5 317 
5.6 


Notes: Estimates are adjusted by random assignment blocks and select baseline characteristics (see section titled 
“Impact Model Specifications” for details). Standard errors are reported in braces. The “p-value difference in 
effects” column presents the p-value for a test of variation in treatment effects among the categories shown for each 
subgroup, based on the H, statistic as described in Greenberg, Meyer, and Wiseman (1994). 

* Students shown as not having a high school diploma include those who earned no degrees, those who earned 
a General Educational Development (GED) certificate, and those who are missing degree information. Students 
shown as having a high school diploma are those who earned a high school diploma, an occupational or technical 
certificate, or another, unspecified higher degree. 


Sources: MDRC calculations from a Baseline Survey, CUNY Institutional Research Database (IRDB), and National 
Student Clearinghouse data 


Speculating Why CUNY’s ASAP Works.—This evaluation was designed to 
estimate the effect of CUNY’s ASAP—a bundled package. The evaluation was not 
designed to disentangle the effectiveness of the individual program components or 
to determine which components matter most. Here, we offer speculative commen- 
tary regarding what drove the large estimated effects. 

ASAP’s full-time requirement and strong messaging to enroll during interses- 
sions (summer/winter), coupled with sufficient supports to facilitate that enroll- 
ment, were probably central to ASAP’s success. Starting in semester one, ASAP 
had a large positive effect on full-time enrollment and winter/summer enrollment.? 
Through six years, the program caused students to attempt an additional ten credits 
and earn an additional seven credits. It seems unlikely ASAP’s effects would have 
been nearly as large without the requirement and encouragement to attempt these 
additional credits. 

That said, it is one thing to require full-time enrollment and encourage winter / 
summer enrollment; it is another to do so while covering students’ tuition, books, 
and transportation, and providing enhanced services to support students to help 
them achieve this goal. It remains unknown how much support is necessary to yield 
substantial effects on full-time and intersession enrollment, and for those effects to 
translate into credits earned. It is possible that at least some additional supports are 
necessary. An ongoing RCT at ten colleges in Ohio focuses solely on encouraging 


20 See Scrivener et al. (2015) for winter/summer enrollment effect estimates, which represent 28 percent of the 
total impact on credits earned through 3 years. 
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additional summer enrollment at a low-cost with minimal additional supports—this 
may provide further insight into this question.?!) Similarly, increasingly popular “15 
to finish” campaigns, which encourage students to attempt 15 credits per semester, 
may shed light on whether getting students to attempt more credits alone can make a 
difference, or if unintended negative consequences may outweigh potential benefits, 
as some have cautioned (Goldrick-Rab 2016). 

In addition to the full-time requirement and encouragement to enroll in summer, 
ASAP requires students to participate in various support services—services that 
students report to appreciate and find helpful. Students had to meet with an advisor 
twice per month, a career and employment specialist once per semester, and attend 
tutoring while taking developmental courses. To encourage students to meet these 
requirements, ASAP linked receipt of a monthly transportation pass with participa- 
tion in these services. The intelligent integration of these components is a critical 
characteristic of ASAP—advising, career guidance, and tutoring are already avail- 
able at most community colleges (including those in this study), but ASAP’s intru- 
sive, monitored, and incentivized approach led to an exceptional service contrast 
with respect to usage of these services. 

In order to link incentives to services, measure articulated service participation 
benchmarks, and track student-staff engagement, ASAP staff record students’ ser- 
vice usage through a management information system (MIS). College ASAP direc- 
tors and the CUNY Central ASAP management team use this information (along 
with administrative /academic data) regularly to monitor program implementation, 
identify areas for improvement, distribute MetroCards appropriately, and measure 
movement toward graduation goals. This type of MIS is essential for the integration 
of services and contributed to the strong implementation of ASAP. 

Finally, ASAP’s effects would not be nearly as large without the advising, which 
students and staff report are central to ASAP’s success. In addition to student sur- 
vey findings, quantified evidence also supports this conclusion. Most notably, at the 
3-year mark, ASAP’s 18 percentage point effect on graduation rates is 7 percentage 
points higher than its estimated effect on earning 60 or more college-level credits, 
the number of credits typically required to earn an associate’s degree. This differ- 
ence suggests ASAP either led more students to take the right combination of credits 
to earn a degree or prompted students to take the final steps necessary, such as filing 
paperwork, to make a degree official. In either case, the ASAP advisor would have 
played a central role. 

No one or two of ASAP’s components alone are likely to yield effects of the 
magnitude estimated in this study. In fact, many of ASAP’s components (or con- 
ceptually similar components) have been evaluated via one or more RCT in com- 
munity college settings, and none have found overall average ITT effect estimates 
of this magnitude. What makes ASAP unique is that its multiple, integrated, and 
well-implemented services address multiple prevalent barriers to student success, 
and those services are offered for three full years. 


?1 See https://www.mdrc.org/publication/can-we-boost-college-summer-enrollment-using-behavioral-science. 
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Generalizability of Findings——A major strength of the present evaluation is 
that the randomized design allows for an unbiased estimator of the causal effect of 
CUNY ASAP. As is true of most randomized trials, a limitation is that statistical 
inference beyond the study sample is difficult to justify since sites (and study par- 
ticipants) were not randomly selected from a population. In this case, generalization 
beyond the study sample requires primarily /ogical inference and uncertainty that is 
difficult to quantify. 

Despite this limitation, we believe there is evidence that ASAP can work for a 
variety of types of students, in multiple settings, and at a large scale. 

With respect to students—tecall that we examined 10 different subgroups (e.g., 
gender, race, age) comprised of 23 types of students (e.g., female, male, black, 
Hispanic, white). For all student types, ASAP’s estimated effect on three-year 
graduation rates was positive, substantial, and (for all but two) statistically sig- 
nificant. This suggests that this intervention can work for a broad array of student 
types. 

With respect to settings—this study took place at 3 distinct colleges and effect 
estimates on graduation rates at all 3 colleges are positive (statistically significant at 
the 2 colleges with larger sample sizes, and an estimated 6 percentage point effect at 
the third college with a smaller sample size), and there is no discernable variation in 
effects across the colleges. This provides prima facie evidence that ASAP can work 
in more than one college, albeit all located in New York City and with centralized 
administration. 

Furthermore, in 2014, MDRC and CUNY partnered to launch the ASAP 
Demonstration in Ohio, an effort to determine whether CUNY ASAP can be 
implemented successfully at three community colleges in Ohio and achieve simi- 
larly positive academic impacts as in New York City. As part of the demonstration, 
Cincinnati State Technical and Community College, Cuyahoga Community College, 
and Lorain County Community College each began operating their own programs 
based on the CUNY ASAP model. The colleges received technical assistance from 
CUNY, as well as operational support from MDRC and the Ohio Department of 
Higher Education. A randomized controlled trial is being conducted at the three 
colleges and early findings show that the Ohio programs substantially increased 
first-semester, full-time enrollment and credit accumulation, as well as persistence 
and full-time enrollment in the second semester (Sommo and Ratledge 2016). The 
early impact estimates are comparable in magnitude to what was observed in the 
present evaluation, adding to the evidence that this program can be implemented in 
a variety of settings and have large effects. Outcomes will continue to be tracked for 
at least three-years in Ohio. 

With respect to scale—since 2010, the time of entry of the two random assign- 
ment cohorts discussed in the present paper, CUNY has expanded ASAP dra- 
matically. The program grew from operating in six colleges to operating in nine 
colleges. In 2017-2018, CUNY reports that ASAP served over 21,000 students, 
with a goal of expanding to 25,000 students per year by 2018-2019. During this 
expansion, the program model described in this paper has changed some: it now 
includes all students regardless of financial aid status, it has created a triage advis- 
ing model in which students visit advisers based on need, and it has eliminated 
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the ASAP seminar? Despite the large expansion and program changes, CUNY’s 
internal analyses find that the three-year graduation rates of students in ASAP 
remain high. In fact, CUNY’s internal data tracking show that the three-year 
graduation rates during the years of the RCT evaluation were lower than for all 
preceding and subsequent cohorts (through the fall 2014 cohort) at the three col- 
leges in the study. 

Beyond New York City and the three locations in Ohio, two additional 
colleges—Skyline College in San Bruno, California, and Westchester Community 
College, part of the State University of New York (SUNY) system—were recently 
awarded grants to receive technical assistance from CUNY to implement programs 
based on CUNY ASAP. The colleges have begun planning efforts and intend to 
start operating their programs by fall 2018, following two years of technical assis- 
tance from CUNY ASAP. MDRC will conduct a randomized controlled trial to eval- 
uate Westchester’s program. Together, this series of experimental evaluations will 
continue to inform whether CUNY’s lauded program can be implemented in other 
contexts and achieve similarly impressive effects. 


Cost and Benefits —While ASAP’s effects are large, they must be considered 
in the context of the program’s cost and benefits. During the 3-year program, we 
estimated the direct cost of the program offer to be around $4,700 per student per 
year, or $14,000 over 3 years (see Scrivener et al. 2015 for details), a 54 percent 
increase over the control group base cost P34 According to CUNY, these costs have 
since come down to closer to $3,450 per student per year in fiscal year 2018, 
owing to economies of scale and program modifications, which were described 
above. Rough estimates in Ohio place the program’s direct costs at around $3,000 
per student per year. 

To place ASAP’s direct cost per student into a broader context, it is 
useful to consider ASAP’s direct costs relative to average expenditures per 
full-time-equivalent student at various types of degree-granting institutions 
throughout the United States. The expenditures per full-time-equivalent student 
at public 2-year institutions are about $13,300 per year. In contrast, total expen- 
ditures per full-time-equivalent student are around $19,100 at private nonprofit 
2-year colleges, $39,100 at public 4-year colleges, and $50,000 at private non- 
profit 4-year colleges.?4 Thus, while ASAP’s direct program costs represent a 
large increase in spending, when added to the base, expenditures remain lower 
than at private nonprofit two-year colleges and are less than half of those of 
four-year public or private nonprofit colleges. From this perspective, the costs 


?2For more detail on the program model, see Boykin and Prince (2015). 

3 The direct program costs were calculated using CUNY’s expenditure data and do not include indirect costs 
associated with the program’s effects on enrollment and credits attempted. Control group base cost was calculated 
by multiplying the cost per credit by the number of credits attempted per control group member. The cost per credit 
was estimated by dividing the college’s annual total expenses and deductions by total instructional activity (credit 
and contact hours attempted). From a social cost perspective, it’s important to note that some of ASAP’s costs are 
transfers (e.g., tuition subsidies) and others are real social cost (advising). 

?4Source: US Department of Education, National Center for Education Statistics, Integrated Postsecondary 
Education Data System (IPEDS), Spring 2008 through Spring 2014, Fall Enrollment component; and Spring 2009 
through Spring 2015, Finance component. (This table was prepared in November 2015.) 
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are relatively low. Nonetheless, from the perspective of cash-strapped community 
colleges, this additional investment is a substantial barrier to considering imple- 
menting CUNY’s program. 

Another approach to considering program cost is to calculate the cost per addi- 
tional outcome, as is done by Carrell and Sacerdote (2017), following Dynarski, 
Hyman, and Schanzenbach (2013). Their focus is on the cost per additional stu- 
dent induced into college (graduation effect estimates are not presented, presum- 
ably due to the shorter follow-up). We consider the cost per graduate, the primary 
outcome of interest in this study. ASAP spent around $14,100 per student to induce 
18 percentage points more students to graduate within 3 years. This translates to 
a cost of roughly $78,000 ($14,100/0.18) per additional degree within 3 years. 
Due to a dearth of studies that estimate program effects on three- or six- degree 
completion and estimate program costs, there is little to compare these estimates 
with.?> 

When considering ASAP’s costs (some of which are transfers 24 like the tuition 
waiver and textbook voucher, and others of which are real social costs, like the 
enhanced advising and tutoring), it is also important to consider the economic 
and social benefits of the program, beyond the direct academic benefits described 
above. Levin and Garcia (2017) monetize and project such benefits, relying on lit- 
erature on the estimated returns to an associate degree. Monetized benefits include 
potentially greater earnings for students, increased tax revenues, reduced public 
health spending, reduced use of the criminal justice system, and reduced use of 
public assistance. 

Levin and Garcia (2017) consider costs and benefits from the perspective of the 
student and the taxpayer. Since ASAP serves students, the student perspective is of 
obvious importance. The taxpayer perspective is also worth consideration because 
CUNY’s ASAP is mainly funded from public sources; moreover, Levin and Garcia 
(2017) note that “On the basis of the program’s promise ... CUNY considered the 
expansion of the program ... To guide this decision, it was necessary to ascertain 
whether ASAP was a good investment for the taxpayer by examining whether the 
benefits of investing in ASAP exceeded the costs.” From the taxpayer perspective, a 
program that benefits its clients and has monetary benefits that exceed the monetary 
costs to the taxpayer is usually a wise investment. Importantly, the taxpayer is not 
just trying to make a profit, they are trying to increase social welfare and a program 
like ASAP could be “worth it” even if the monetary costs to taxpayers exceed the 
monetary benefits. 

Projecting the net monetary benefit to taxpayers requires strong assumptions and 
the estimates are imprecise. Nonetheless, in an attempt to consider the net benefit 
to taxpayers, we can utilize Levin and Garcia’s (2017) estimate of the taxpayer 


?5 Moreover, we are cautious about over interpreting cost per outcome across studies. If a payee (e.g., govern- 
ment, society, etc.) is willing to pay a large amount for additional degrees, then they may be happy to pay more per 
degree in order to get a lot of additional degrees, rather than pay less per degree to get only a few marginal degrees. 
That said, with a fixed budget for new programs and many eligible participants, the cost per outcome framework 
can be a useful way to make decisions on whether to serve more students with a less expensive program or fewer 
students with a more expensive program. 

26 Transfers, including the tuition waiver, textbook voucher, and monthly MetroCards, account for about 
24 percent of ASAP’s direct costs. 


VOL. 11 NO.3 ~~ WEISS ET AL.: LONG-TERM RESULTS OF A RANDOMIZED TRIAL OF CUNY’S ASAP 283 


benefit per additional associate’s degree ($205,500 per additional associate degree), 
Scrivener et al.’s (2015) estimate of the net cost of providing ASAP ($14,000 per 
student), and the 10 percentage point estimated effects on graduation rates pre- 
sented here. Applying this “back-of-the envelope” analysis, the total net benefits 
of ASAP are around $7,300 per student higher than the net benefits for the con- 
trol group?” Applying a 90 percent confidence interval to the estimated effect 
on 6-year graduation rates yields a range of net benefits from negative $6,000 to 
positive $16,000. This range does not consider the uncertainty associated with the 
estimated benefits projections, which would widen the confidence intervals con- 
siderably ?® In sum, taxpayers likely recoup some, if not all the cost of paying for 
ASAP, in addition to the increased social welfare for the served students. It will 
be important to see how these analyses might change—or remain steady—with 
longer-term follow-up. In the future, we hope to estimate ASAP’s effects on observed 
labor market outcomes, rather than projections, which will shed more light on 
this issue. 

From the student perspective ASAP is highly cost-beneficial under weak 
assumptions. This is because the cost for program group students is lower than 
the cost for control students due to the savings associated with ASAP’s tuition 
waiver, textbook voucher, and monthly transportation pass.P)| Thus, the pro- 
jected benefits for students, which rely on strong assumptions regarding the 
projected returns to a degree, only add to the already positive net benefits for 
ASAP students. 

CUNY ASAP is perhaps the most effective college-completion program in 
higher education to be rigorously evaluated through a randomized experiment. 
This evaluation shows that with the right combination of long-lasting supports, 
requirements, and messages, it is possible to dramatically increase graduation 
rates at community colleges. 


°7Calculated as $205,514 x 0.104 — $14,028 = $7,345. 

?8Levin and Garcia (2017) state that their taxpayer benefit per associate’s degree is “conservative” (meaning 
downward biased), which could increase the net benefits. Counterbalancing this, we assume that the 10 percentage 
point graduation effects will be maintained, which likely results in upward bias, given the observed trend in esti- 
mated effects on graduation. There is likely additional uncertainty because our basic analyses do not account for 
the estimate that 41 percent of the ASAP sample earned a degree, on average, | year earlier than they would have, 
which would increase the net benefit somewhat. They also do not account for the fact that Levin and Garcia’s (2017) 
taxpayer benefit was derived for 3-year graduation rates for 23-year-olds, rather than 6-year graduation rates for 
26-year-olds, which would decrease the net benefits somewhat. 

?°While there may be greater opportunity cost for students in ASAP, the one-year follow-up survey does not 
find clear evidence that control group members are working at a higher rate than program group members. 
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APPENDIX A 


TABLE Al—ADDITIONAL CHARACTERISTICS OF SAMPLE MEMBERS AT BASELINE 


JULY 2019 


Control Estimated 


Characteristic (percent) mean difference | Observations 
Highest grade completed 
10th grade or lower 79 —1.2 896 
17 
11th grade 72. 1.2 896 
1.8 
12th grade* 75.7 0.4 896 
2.9 
Missing 9.2 —0.4 896 
1.9 
Date of high school graduation/GED receipt 
During the past year 47.4 4.0 896 
3.3 
Between one and two years ago 13.9 -14 896 
23 
Between two and five years ago 13.5 —0.8 896 
23 
More than five years ago 13.3 —0.2 896 
2.3 
Has not earned a diploma/GED* 6.3 —0.4 896 
1.6 
Missing 5.6 —1.2 896 
1.5 
Student’s status 
Incoming freshman 57.3 5.9 896 
3.0 
Returning student Pall —3.6 896 
2.8 
Transfer student 7.6 —2.4 896 
1.6 
Highest degree student plans to attain 
Associate’s 2.6 0.4 863 
1.1 
Bachelor’s 30.2 2.3 863 
3.2) 
Master’s 41.9 —0.6 863 
3.4 
Professional or doctorate 18.3 —0.8 863 
2.6 
Beyond an associate’s, unspecified 7.0 —13 863 
iA 


(continued) 


VOL. 11 NO.3 WEISS ET AL.: LONG-TERM RESULTS OF A RANDOMIZED TRIAL OF CUNY’S ASAP 285 


TaBLE Al—ADDITIONAL CHARACTERISTICS OF SAMPLE MEMBERS AT BASELINE (continued) 


Control — Estimated 
Characteristic (percent) mean difference | Observations 


Mother’s educational attainment 


Not a high school graduate 20.4 —1.9 896 
2.6 
High school diploma or GED 22.0 —0.5 896 
2.8 
Some college, no degree 16.0 0.0 896 
2.5 
College degree (AA, BA, MA, PhD) 18.9 —0.4 896 
2.6 
Missing 22.7 2.8 896 
2.8 
Language other than English spoken regularly at home 44.2 1.0 885 
3:3 
Lives with parents 73.6 0.3 872 
3.0 
Parents pay > 1/2 of expenses 
Yes 40.2 1.5 896 
33 
Missing 18.9 -19 896 
2.6 
Marital status 
Married 6.3 —0.2 896 
1.6 
Missing 14.6 1.4 896 
2.4 
Has one or more children 13.1 4.3 884 
2.4 
Number of hrs worked per week, among employed 
1-10 hours 6.7 2.9 259 
3.4 
11-20 hours 31:4 6.7 259 
5.9 
21-30 hours 39.3 —16.1 259 
5.8 
31-40 hours 20.7 7.8 259 
53 
> 40 hours 2.2 —1.3 259 
15 


Notes: Missing values are only included in variable distributions for characteristics with more than 5 percent of the 
sample missing. Estimated differences are adjusted by random assignment blocks only. Standard errors are reported 
in brackets. 

* This number includes students who were enrolled in high school at study intake. 


Source: MDRC calculations from a Baseline Survey 
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TABLE A2—ENROLLMENT AT ANY COLLEGE BY SEMESTER 


Estimated effect 


Control Covariate 
mean adjusted Unadjusted Observations 
Percent of students enrolled 

Semester | 94.2 2.2 2.5 896 
1.4 1.4 

Semester 2 81.6 9.9 10.3 896 
2.3 2) 

Semester 3 70.3 6.8 7.0 896 
3.0 2.9 

Semester 4 62.9 7S 8.3 896 
3:1 3.1 

Semester 5 55.1 6.1 6.8 896 
3.3 3.3 

Semester 6 47.4 3.8 4.1 896 
3.4 3.3 

Semester 7 (post-program) 42.5 —3.0 —2.4 896 
3.3 3.3 

Semester 8 (post-program) 40.9 —3.9 —3.2 896 
3.3 3.3 

Semester 9 (post-program) 38.2 —5.0 —4.5 896 
3.2 3.2 

Semester 10 (post-program) 35.1 —2.0 -1.8 896 
3.2 3.2 

Semester 11 (post-program) 31.9 —4.5 —4.0 896 
3.1 3.1 

Semester 12 (post-program) 26.5 —2.6 —2.0 896 
2.9 29) 

Number of terms enrolled 6.3 0.2 0.2 896 
(3.5) 0.2 0.2 


Notes: Covariate adjusted estimates are adjusted by random assignment blocks and select baseline characteristics 
(see section titled “Impact Model Specifications” for details). Unadjusted estimates are adjusted by random assign- 
ment blocks only. Standard deviations for continuous outcomes are reported in parantheses. Standard errors are 
reported in brackets. Enrollment is based on courses that students are enrolled in at the end of the add/drop period. 


Sources: MDRC calculations from CUNY Institutional Research Database (IRDB) and National Student 
Clearinghouse data 
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TABLE A3—FULL-TIME ENROLLMENT AT ANY CUNY COLLEGE 


Estimated effect 


Control -Covariate 


mean adjusted Unadjusted Observations 
Percent of students enrolled full time 

Semester 1 85.2 10.2 10.6 896 
1.9 1.9 

Semester 2 65.2 19.8 20.4 896 
2.8 2.8 

Semester 3 59.6 14.0 14.2 896 
3.1 3.1 

Semester 4 47.9 14.7 15.1 896 
3.3 3.3 

Semester 5 39.8 5.5 5.9 896 
3.3 3.3 

Semester 6 27.4 3.5 5.4 896 
3.0 3.0 

Semester 7 (post-program) 22.5 3.5, 3.9 896 
2.8 2.9 

Semester 8 (post-program) 22.5 1.2 1.5 896 
2.8 2.8 

Semester 9 (post-program) 21.1 —0.8 —0.5 896 
2.7 2.7 

Semester 10 (post-program) 18.2 —0.4 0.4 896 
Qf 2.6 

Semester 11 (post-program) 16.4 —2.5 —2.0 896 
2.4 2.4 

Semester 12 (post-program) 11.7 —0.3 0.3 896 
2:2 ace 


Notes: Covariate adjusted estimates are adjusted by random assignment blocks and select baseline characteristics 
(see section titled “Impact Model Specifications” for details). Unadjusted estimates are adjusted by random assign- 
ment blocks only. Standard errors are reported in brackets. Enrollment is based on courses that students are enrolled 
in at the end of the add/drop period. 


Source: MDRC calculations from CUNY Institutional Research Database (IRDB) 
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TABLE A4— MARGINAL CREDITS ATTEMPTED AND EARNED AT CUNY COLLEGES 


Estimated effect 
Control Covariate 
mean adjusted Unadjusted Observations 
Panel A. Credits attempted 
Semester | 13.91 2.09 2.18 896 
(5.04 0.32 0.32 
Semester 2 11.23 3.18 3.28 896 
(6.50 0.41 0.41 
Semester 3 O72. 2.50 2.56 896 
(7.37 0.50 0.49 
Semester 4 8.31 71 1.86 896 
(7.58 0.51 0.51 
Semester 5 6.71 0.67 0.78 896 
(7.25 0.48 0.48 
Semester 6 4.90 0.82 0.83 896 
(6.49 0.45 0.45 
Semester 7 (post-program) 4.09 0.30 0.35 896 
(6.05 0.41 0.42 
Semester 8 (post-program) 4.03 —0.11 —0.08 896 
(6.22 0.41 0.42 
Semester 9 (post-program) 3.67 —0.33 —0.26 896 
(5.87 0.39 0.39 
Semester 10 (post-program) 3.31 —0.17 —0.06 896 
(5.91 0.40 0.39 
Semester 11 (post-program) 2.84 —0.42 —0.30 896 
(5.45 0.36 0.36 
Semester 12 (post-program) 2.20 —0.20 —0.08 896 
(4.86 0.33 0.33 
Cumulative credits attempted 74.94 10.04 11.05 896 
(47.36 3.08 3.08 


(continued) 


VOL. 11 NO.3 WEISS ET AL.: LONG-TERM RESULTS OF A RANDOMIZED TRIAL OF CUNY’S ASAP 289 


TABLE A4d— MARGINAL CREDITS ATTEMPTED AND EARNED AT CUNY COLLEGES (continued) 


Estimated effect 
Control Covariate 
mean adjusted Unadjusted Observations 
Panel B. Credits earned 

Semester 1 9:31 1.95 2.08 896 
(6.38 0.40 0.40 

Semester 2 7.86 yan | 2.25 896 
(6.69 0.45 0.45 

Semester 3 7.19 1.63 1.73 896 
(6.77 0.46 0.46 

Semester 4 6.06 1.39 1.56 896 
(6.72 0.46 0.46 

Semester 5 4.93 0.48 0.55 896 
(6.13 0.41 0.41 

Semester 6 3.84 0.50 0.51 896 
(5.62 0.39 0.39 

Semester 7 (post-program) 3.31 0.16 0.22 896 
(5.37 0.36 0.37 

Semester 8 (post-program) 3.18 —0.08 —0.05 896 
(5.42 0.37 0.37 

Semester 9 (post-program) 3.05 —0.38 —0.29 896 
(5.24 0.35 0.35 

Semester 10 (post-program) 2.61 —0.05 0.06 896 
(5.09 0.36 0.35 

Semester 11 (post-program) 2.32 —0.46 —0.34 896 
(4.74 0.31 0.31 

Semester 12 (post-program) 1.76 —0.27 —0.15 896 
(4.27 0.29 0.28 

Cumulative credits earned 55.07 7.00 8.14 896 
(45.84 2.98 3.00 


Notes: Covariate adjusted estimates are adjusted by random assignment blocks and select baseline characteristics 
(see section titled “Impact Model Specifications” for details). Unadjusted estimates are adjusted by random assign- 
ment blocks only. Standard errors are reported in brackets. Standard deviations for continuous outcomes are shown 
in parantheses. Measures of cumulative credits earned exclude courses that are passed more than once. 


Source: MDRC calculations from CUNY Institutional Research Database (IRDB) 
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TABLE AS—CUMULATIVE CREDITS ATTEMPTED AND EARNED AT CUNY COLLEGES 
Estimated effect 
Control Covariate 
mean adjusted Unadjusted Observations 
Panel A. Cumulative credits attempted 
Semester | 13.91 2.09 2.18 896 
(5.04) 0.32 0.32 
Semester 2 25.15 5.26 5.46 896 
(9.93) 0.63 0.62 
Semester 3 34.87 7.76 8.01 896 
(15.21) 0.99 0.97 
Semester 4 43.18 9.A7 9.88 896 
(20.81) 1.35 1.34 
Semester 5 49.89 0.14 10.65 896 
(25.58) 1.66 1.65 
Semester 6 54.80 0.97 11.49 896 
(29.30) 1.92 1.90 
Semester 7 (post-program) 58.89 1.26 11.83 896 
(32.67) 2.15 2.14 
Semester 8 (post-program) 62.92 1.15 11.75 896 
(36.16) 2.38 2.38 
Semester 9 (post-program) 66.59 0.82 11.50 896 
(39.58) 2.60 259 
Semester 10 (post-program) 69.90 0.65 11.43 896 
(43.05) 2.81 2.81 
Semester 11 (post-program) 72.74 0.24 11.13 896 
(45.47) 2.96 2.96 
Semester 12 (post-program) 74.94 0.04 11.05 896 
(47.36) 3.08 3.08 
Panel B. Cumulative credits earned 
Semester | 9.31 1.92 2.05 896 
(6.38) 0.40) 0.40 
Semester 2 17.14 4.04 4.31 896 
(11.87) 0.75 0.75 
Semester 3 24.32 5.65 6.03 896 
(17.24) 1.12 1.12 
Semester 4 30.36 7.03 7.58 896 
(22.27) 1.45 1.45 
Semester 5 35:23 7.54 8.15 896 
(26.42) 1.72 1.72 
Semester 6 39.02 8.05 8.67 896 
(29.86) 1.95 1.95 
Semester 7 (post-program) 42.29 8.20 8.88 896 
(33.06) 2.16 2.17 
Semester 8 (post-program) 45.44 8.14 8.84 896 
(36.24) 2.38 2.39 
Semester 9 (post-program) 48.46 7.74 8.53 896 
(39.43) 2.58 2.60 
Semester 10 (post-program) 51.04 7.72 8.61 896 
(42.31) 277 2.78 
Semester 11 (post-program) 53.31 7.30 8.31 896 
(44.38) 2.89 2.90 
Semester 12 (post-program) 55.07 7.00 8.14 896 
(45.84) 2.98 3.00 


Notes: Covariate adjusted estimates are adjusted by random assignment blocks and select baseline characteristics 
(see section titled “Impact Model Specifications” for details). Unadjusted estimates are adjusted by random assign- 
ment blocks only. Standard errors are reported in brackets. Standard deviations for continuous outcomes are shown 
in parantheses. Measures of cumulative credits earned exclude courses that are passed more than once. 


Source: MDRC calculations from CUNY Institutional Research Database (IRDB) 
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TABLE A6—DEGREE COMPLETION AT ANY COLLEGE 


Estimated effect 


Control Covariate 
mean adjusted Unadjusted Observations 
Earned any degree (percent) 
Semester 1 0.0 0.0 0.0 896 
Semester 2 0.0 0.2 0.2 896 
0.2 [0.2] 
Semester 3 1.1 1.8 1.7 896 
1.0 [0.9] 
Semester 4 8.8 5.6 6.0 896 
2.2 [2.1] 
Semester 5 15.3 13.4 14.2 896 
27 [2.7] 
Semester 6 22.0 17.7 18.3 896 
3.1 [3.0] 
Semester 7 (post-program) 26.3 15.7 16.7 896 
3.1 [3.1] 
Semester 8 (post-program) 29.9 15.3 16.0 896 
3.2 3.2 
Semester 9 (post-program) 32.4 14.2 14.9 896 
3:2) 3.2) 
Semester 10 (post-program) 36.4 11.8 12.4 896 
3.3 3.3 
Semester 11 (post-program) 38.2 11.5 12.4 896 
3.3 3.3 
Semester 12 (post-program) 40.4 10.4 11.5 896 
3.3 3.3 
Earned an associate’s degree (percent) 37.1 12.2 13.0 896 
3.3 3.3 
Earned a bachelor’s degree or higher (percent) 16.2 0.5 1.2 896 
2.5 2.5 
Highest degree earned (percent) 
Certificate 0.0 0.0 0.0 896 
0.0 0.0 
Associate’s 23.8 9.8 10.3 896 
3.0 3.0 
Bachelor’s or higher 16.2 0.5 1.2 896 
29 2.5 


Notes: Covariate adjusted estimates are adjusted by random assignment blocks and select baseline characteristics 
(see section titled “Impact Model Specifications” for details). Unadjusted estimates are adjusted by random assign- 
ment blocks only. Standard errors are reported in brackets. Degree receipt is cumulative. Those who earned a degree 
in an earlier semester are counted as having a degree in subsequent semesters. 


Sources: MDRC calculations from CUNY Institutional Research Database (IRDB) and National Student 


Clearinghouse data 
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TABLE A7—ENROLLMENT BY COLLEGE TYPE AND SEMESTER 
Estimated effect 
Control Covariate 
mean adjusted Unadjusted Observations 

Panel A. Enrolled at any 2-year college ( percent) 

Semester | 93.5 2.7 2.9 896 
15 1.4 

Semester 2 80.9 10.1 10.5 896 
2.3 2.3 

Semester 3 68.8 6.9 7.2 896 
3.0) 3.0) 

Semester 4 58.0 8.0 8.6 896 
3.3 3.2) 

Semester 5 43.8 1.2 i 896 
3.3 3.3 

Semester 6 30.1 —3.1 —3.5 896 
3.0) 3.0) 

Semester 7 (post-program) 20.0 —7.8 -7.9 896 
25 24 

Semester 8 (post-program) 15.5 —6.8 —6.9 896 
2.2 2.2 

Semester 9 (post-program) 11.5 —3.6 —3.5 896 
2.0 2.0 

Semester 10 (post-program) 79 —2.0 -17 896 
ey Ly 

Semester 11 (post-program) 6.1 0.7 1.0 896 
1.7 17. 

Semester 12 (post-program) 2.2 0.0 —0.1 896 
ile) 15 

Panel B. Enrolled at any 4-year college (percent) 

Semester 1 0.7 —0.5 —0.4 896 
0.5 0.4 

Semester 2 0.9 —0.2 —0.2 896 
0.6 0.6 

Semester 3 1.6 0.0 0.0 896 
0.8 0.8 

Semester 4 3:2 —0.1 0.2 896 
135 15 

Semester 5 12.4 43 4.8 896 
24 2.3 

Semester 6 17.5 7.0 7.6 896 
2.7 2d 

Semester 7 (post-program) 22.5 48 ie] 896 
29 2.9 

Semester 8 (post-program) 25.4 2.9 3.7 896 
3.0 3.0) 

Semester 9 (post-program) 26.7 —0.4 —0.1 896 
3.0) 3.0) 

Semester 10 (post-program) 27.2 —0.1 —0.1 896 
3.0) 3.0) 

Semester 11 (post-program) 25.8 —5.0 —4.8 896 
2.8 2.8 

Semester 12 (post-program) 21.3 —2.4 -17 896 
27 27 


Notes: Estimates are adjusted by random assignment blocks and select baseline characteristics (see section titled 
“Impact Model Specifications” for details). Standard errors are reported in brackets. Enrollment is based on courses 


that students are enrolled in at the end of the add/drop period. 


Sources: MDRC calculations from CUNY Institutional Research Database (IRDB) and National Student 


Clearinghouse data 
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TABLE A8—DEGREE COMPLETION AT ANY COLLEGE BY DEGREE TYPE 


Estimated effect 


Control Covariate 
mean adjusted Unadjusted Observation 

Panel A. Earned an associate's degree (percent) 

Semester 1 0.0 0.0 0.0 896 

Semester 2 0.0 0.2 0.2 896 
0.2) 0.2 

Semester 3 Ll 1.8 1.7 896 
1.0 0.9 

Semester 4 8.8 5.6 6.0 896 
2.2 Zl 

Semester 5 15.3 3.4 14.2 896 
Qu 2.7 

Semester 6 21.8 7.9 18.5 896 
3.0 3.0 

Semester 7 (post-program) 26.1 5.8 16.7 896 
3.1 3.1 

Semester 8 (post-program) 29.4 5:2 15.8 896 
3.2) 3.2 

Semester 9 (post-program) 31.7 44 14.9 896 
3.2) 3.2 

Semester 10 (post-program) 34.8 2.0 12.4 896 
33 33 

Semester 11 (post-program) 35.7 12.4 13.1 896 
3.3 3.3 

Semester 12 (post-program) 37.1 12:2 13.0 896 
3.3 3.3 

Panel B. Earned a bachelor’s degree or higher (percent) 

Semester 1 0.0 0.0 0.0 896 

Semester 2 0.0 0.0 0.0 896 

Semester 3 0.0 0.0 0.0 896 

Semester 4 0.0 0.0 0.0 896 

Semester 5 0.0 0.2 0.2 896 
0.2 0.2 

Semester 6 0.0 0.7 0.7 896 
0.4 0.4 

Semester 7 (post-program) 0.7 0.4 0.6 896 
O7 0.7 

Semester 8 (post-program) 2.0 1.9 2.2 896 
12 1.2 

Semester 9 (post-program) 3.8 2.8 3:3 896 
1.5 15 

Semester 10 (post-program) 8.1 3.1 3.7 896 
2.0 2.0 

Semester 11 (post-program) 12.4 1:7 2.3 896 
2.3 2.3 

Semester 12 (post-program) 16.2 0.5 1.2 896 
25 25 


Notes: Estimates are adjusted by random assignment blocks and select baseline characteristics (see section titled 
“Impact Model Specifications” for details). Standard errors are reported in brackets. Degree receipt is cumulative. 
Those who earned a degree in an earlier semester are counted as having a degree in subsequent semesters. 


Sources: MDRC calculations from CUNY Institutional Research Database (IRDB) and National Student 
Clearinghouse data 
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